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SYSTEM AND METHOD FOR HIGH-RESOLUTION IMAGING 
WITH A SMALL-FORMAT FOCAL-PLANE ARRAY USING 

SPATIAL MODULATION 

CROSS-REFERENCE TO RELATED APPLICATION 

The present application claims the benefit of U.S. Provisional Patent Application 
Serial No. 60/456,836, which was filed on March 21 . 2003 by John Merchant for fflGH- 
RESOLUTION IMAGING WITH A SMALL-FORMAT FOCAL-PLANE ARRAY 
USING SPATIAL MODULATION, and is hereby incorporated by reference. 

STATEMENT REGARDING FEDERALLY SUPPORTED 
RESEARCH OR DEVELOPMENT 

This invention was made with government support under Air Force Contract No. 

F33615-99-C-1422. The government has certain rights in this invention. 

BACKGROUND OF THE INVENTION 

Field of the Invention 

This invention relates generally to the detection of unresolved point targets in 
clutter using a Spatial Modulation staring array image sensor that could incorporate Fo- 
veal Enhanced Imaging. 

Background Information 

Detection of unresolved point targets in clutter using an imaging electro optical 
(EO) or infrared (IR) sensor is required by many military and civihan system applica- 
tions — ^for example, a missile threat warning (to protect aircraft from air-to-air and 
ground-to-air missiles), infrared search and track (IRST), long-range ballistic missile 
early launch detection, as well as security/surveillance appUcations in perimeter control, 
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port security, and airport security. The fundamental detection problem in all of these 
systems is the identification of a point target against a background of sensor noise and 
scene clutter. In many cases detection is clutter-limited, not noise-limited. 

It is well known that to achieve a high target-to-clutter ratio, a high-resolution im- 
aging sensor should be used. The power spectral density (PSD) of clutter typically falls 
off rapidly with increasing spatial frequency, whereas the PSD of a point target remains 
constant up to the highest spatial frequency passed by the sensor. The target-to-clutter 
ratio, therefore, generally increases with increasing spatial frequency. A high-resolution 
imaging sensor provides access to high spatial frequencies, thereby enabling a high tar- 
get-to-clutter ratio to be realized. 

The resolution of an imaging sensor is bounded by the angular subtense of an ar- 
ray detector element, employed in the sensor. Providing the high-resolution (e.g., 1 miUi- 
radian (mr)) necessary for a high target-to-clutter ratio can be a particularly significant 
problem when this high-resolution must be provided continually over a wide field of 
view (WFOV) (e.g., 1 .5 r) that is often required. This problem is quantified by the large 
number of pixels involved. (E.g., 1,500 x 1,500 pixels for 1 mr resolution over a 1.5 ra- 
dians (r) field of view (FOV)). 

There are three basic types of imaging sensors that can be used for the detection 
of unresolved point targets: scanning, conventional staring, and Spatial Modulation star- 
ing. Each of these sensor types takes a different approach to the fundamental problem of 
sensing and processing a large number of pixels over the FOV. 

In a scanning sensor, a single detector element or a small array of detector ele- 
ments is scanned, opto-mechanically, over the required FOV, The detector array in a 
scanning sensor will have many fewer elements (e.g., 10x10) than there are pixels over 
the FOV (e.g., 1,500 x 1,500). However, the scanning sensor is undesirable in many ap- 
plications because of the cost and complexity of opto-mechanical scanning systems, and 
also because any particular point in the scene is sensed only periodically, and briefly, as 
the small detector array is repetitively scanned over the scene in a sweeping motion. For 
these and other reasons a staring sensor is employed in many applications. 
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In a conventional staring sensor the entire FOV is imaged at one time onto a focal 
plane array (FPA) of discrete detector elements. Each detector element generates a pic- 
ture element, or "pixel," in the output image that is read out electronically from the FPA 
at a certain frame rate. The conventional staring sensor senses the large number of pixels 
over the object scene by employmg an FPA that has the same large number of detector 
elements. A limitation of the conventional staring sensor is the relatively small format 
(e.g., 256 X 256) of affordable, or even available, FPAs. A conventional staring sensor 
with a small-format FPA cannot provide the high-resolution necessary for high-target-to- 
clutter detection over a WFOV. 

An additional limitation of the conventional staring sensor is the high processing 
load required to process the large number of pixels. A very processing-intensive high- 
pass spatial-filtering operation, or its equivalent, must be performed on the output images 
from the detector FPA. This operation is needed to access the high-end of the spatial fre- 
quency spectrum where the target-to-clutter ratio is highest. The higher the resolution of 
the sensor, not only the higher the number of pixels per frame to process, but also the 
higher the necessary FPA frame rate to avoid motion smear in a dynamic environment. 
Additionally, very processing-intensive te/w/?ora/-filtering operations (such as track- 
before-detect) may be required to achieve maxunum detection sensitivity when the reso- 
lution and frame rate are both high. 

The spatial-modulation staring sensor is a new type of staring sensor for point- 
target detection in clutter. It is able to effectively sense a large number of pixels over the 
object scene using a detector FPA that has many fewer detector elements. High- 
performance point-target detection in clutter (e.g., with 1,500 x 1,500 pixels over the ob- 
ject scene) can thereby be implemented using an affordable and available small-format 
detector FPA (e.g., 256 x 256). Moreover, no spatial filtering of the detector FPA output 
is required, and temporal filtering can be effected by simple frame averaging because of 
the low pixel-to-subtense density of the output image from the FPA, 

Fig. 1 is a schematic diagram of a spatial-modulation staring sensor. A high- 
resolution image 10 of the object scene is formed by high-resolution front-end optics 20. 
This image is formed over a chopping reticle 30 located at an intermediate focal plane 40. 
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The reticle 30 is a uniform checkerboard pattern of transparent and opaque cells. The 
edges of the checkerboard cells define a set of rectangular x, y coordinate axes. Typi- 
cally, the size of each cell is approximately the same as that of the point-spread function 
(PSF) of the front-end optics. 

The input high-resolution image is multiplied, i,e., spatially modulated, by the re- 
ticle pattern. The checkerboard pattern can be characterized mathematically by a spatial 
function that has the value "1" over a clear checkerboard cell and the value "0" over an 
opaque cell. The modulated (reticle-modified) image appearing at the back surface of the 
reticle 30 is then re-imaged by relay optics 50 onto the detector FPA 60. There are ap- 
proximately SMF X SMF as many checkerboard cells covering that image as there are 
detector elements in the FPA, where SMF is a design parameter (e.g., SMF = 6). As 
Fig. 2 illustrates, that is, a subarray 70 of SMF x SMF checkerboard cells (clear 76 and 
opaque 78) is imaged onto each detector element 80 in the FPA 60. 

The reticle 30 is moved steadily in a linear motion along a fixed direction trans- 
verse to the intermediate focal plane. An actuator (not shown, but represented by multi- 
dimension arrows 37) is employed to move the reticle within a fixed line along a two- 
dimensional plane of movement. This fixed direction is inclined to the rectangular axes 
(x, y) so that, for example, when a point on the reticle has moved by four cell widths 
along one axis (e.g., the x axis) it will then have moved by one cell width along the other 
axis (e.g., the;/ axis). The reticle speed is such that it moves 0,5-cell width along one 
axis (e.g., x) and 0. 125 cell width along the other axis (e.g., y) during each FPA frame 
period. 

If a point target is present in a static scene the image of that target on the reticle 
30 will sometimes fall over a clear reticle cell, sometimes over an opaque cell and some- 
times between an opaque and clear cell. Because the checkerboard reticle is moved at 0.5 
cell width per frame a target that falls over a clear cell in frame n will fall mostly over an 
opaque cell in frame w + 2, In that case the target signal can be recovered, and the clutter 
cancelled, by subtracting frame n from frame w + 2. In the general case it can be shown 
that the target signal can be recovered no matter where its image falls over the checker- 
board reticle by similarly processing a sequence of eight frames. 
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As Fig. 3 indicates, processing circuitry 72 receives the sequence of raw output 
frames from the detector FPA 60. Those frames can be symbolized as Vn(x,yX where n is 
the frame number and (x^) are the coordinates of a detector element or output pixel. The 
processing circuitry 72 demodulates this sequence of frames in the following manner to 
derive a sequence of spatial-modulation output frames Wn{xji) in which clutter is 
strongly attenuated but a point target is not: 

The signal from a point target is preserved by this frame-differencing action be- 
cause the target image on the reticle is localized to a cell, and the moving reticle pattern 
effectively complements after every other frame. The target signal will thus appear in 
one or more of the four difference images V\ - F3, V2 - F4, V5 - Vy, Ve - Fg, depending 
upon the position of the target relative to the ceU pattern. It can be shown that the target 
signal appears in FFat approximately its fiiU amplitude no matter where the target falls 
over the reticle pattern. 

However, the background clutter is virtually canceled by this same differencing 
action. This is because the clutter is not localized to a cell width, but extends contmu- 
ously over the reticle cells. Each detector element receives the clutter signal falling over 
an SMF x SMF (e.g., 6x6) subarray of cells. The magnitude of this clutter signal is al- 
most unchanged as the reticle cells complement. Thus the differencing action (e.g., 
V\ - F3), which preserves the target signal, cancels the clutter almost to zero. 

After moving some distance the reticle motion is reversed. After moving some 
distance in this other direction the reticle motion is again reversed, and so on. The size of 
the reticle is large enough so that the image of the object scene is covered by the check- 
erboard pattern for all positions of the reticle as it is moved forward and backward along 
its direction of motion. 

Frame differencing, as described above, provides (in W) a high level of back- 
ground-clutter suppression while preserving the signal from a point target at almost full 
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amplitude. In addition to the spatial-modulation output W, a conventional output VTD 
can be derived by frame addition. For example: 

VIDnixj;) = V„(x,y) + V„^2ix^) 

This conventional output is essentially the same as would be provided by a con- 
ventional staring sensor covering the same FOV using the same FPA. 

Point targets are generally not detectable in the conventional output image VID 
because of its low angular resolution. Point targets can be detected in the spatial-modula- 
tion output image FF because it is derived by spatial modulation from the high-resolution 
optical image formed on the reticle. A detection derived from FFcan be used to annotate 
the conventional output VID presented by, for example, a display 74 to show the position 
of the detected target in relation to the background scene. 

Human vision is very strongly based upon true foveal imaging. A Spatial Modu- 
lation sensor with Foveal Enhanced Imaging mimics the foveal-peripheral vision of the 
eye, achieving high-resolution point target detection over a wide field-of-view (WFOV), 
using a low density (e.g. 256 x 256) detector array, while providing conventional high- 
resolution imaging over a small area, also using a low density detector array, and rapidly 
switching that small area within the total field-of-regard. In human vision the WFOV 
sensmg performed by peripheral vision allows objects of any size to be detected, although 
not recognized. A Spatial Modulation sensor with Foveal Enhanced Imaging performs a 
different low density sampling than human peripheral vision and can detect only unre- 
solved (hot spot) point targets. 

SUMMARY OF THE INVENTION 
Balanced Demodulation 

A spatial-modulation staring sensor is able to perform point target detection with 
a much smaller-format focal plane array FPA than would otherwise be required. Con- 
ventionally, though, the frame rate required for it to perform effectively for rapidly mov- 
ing scenes, such as those observed from a rapidly maneuvering aircraft, can impose a 
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processing sensing and processing load that is often impractical to bear. Unless the frame 
rate is high, the magnitude of the residual clutter increases sharply when the clutter is 
moving with respect to the sensor. One approach to reduce the processing load is to em- 
ploy "balanced-demodulation procedures." A system that employs balanced demodula- 
tion moves a chopping reticle located in the intermediate focal plane where front-end op- 
tics focuses a high-resolution image. The chopping reticle has a checkerboard pattern of 
clear and opaque cells and moves in an uneven rate (e.g. Ax tsy) along the x-axis and y- 
axis. The exemplary chopping reticle has a checkerboard pattern of clear and opaque 
cells and moves in an uneven rate (e.g. Ax tSy) along the x-axis and y-axis. The result- 
ing reticle-modified image is projected on a focal plane array from which differences are 
calculated to generate the desired balanced demodulation value. Using such procedures in 
place of the types of procedures previously employed in spatial-modulation staring sen- 
sors tends to preserve a high level of clutter suppression with scene motion without re- 
quiring an excessive frame rate. An illustrative embodiment, employs such procedures in 
ways that implement infinite-impulse-response filtering, maximizes the use of all output 
frames, and, by summing absolute values rather than taking the square root of the sum of 
the squares, reduce the computation burden. The necessary frame rate can also be limited 
by basing reticle motion on target motion. 

Foveal Enhanced Imaging 

According to ftirther embodiments, the basic spatial-modulation method described 
herein is enhanced. The baseline spatial-modulation imaging (SMI) staring sensor per- 
forms high-resolution point-target acquisition over a WFOV. It also provides a low- 
resolution conventional image of the scene over that WFOV. However, in some cases it 
is desired to provide an operator with a high-resolution view not only of point targets but 
also of all of the scene's spatial-frequency components. For example, six sensors, each 
covering a 90° FOV, could be distributed over an aircraft to detect point targets over a 
FOV that subtends a soUd angle of 4D steradians. It could also be desired to provide an 
operator in that aircraft with the ability to see in any direction, at high-resolution, over 
such a field of view. This is sometimes referred to as a "glass cockpit." 
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The detection and recognition of targets over a WFOV requires high-resolution. 
Conventionally this high-resolution is provided by a high pixel density sensor having a 
correspondingly high density FPA. SMI provides the high-resolution needed for target 
detection, but with many fewer pixels and a correspondingly much smaller FPA, Foveal 
Enhanced Imaging (FBI) is a means of examining a target, detected initially by SMI as an 
unresolved point target (hot spot), by conventional imaging at high pixel density over a 
small FOV at that target. Spatial Modulation plus FEI enables targets to be detected and 
then examined using many fewer pixels (detector elements) than would otherwise be 
needed. 

With basic (or pseudo) FEI the high-resolution pixel density used for narrow field 
of view (NFOV) imaging (for target recognition) is the same as the optical resolution 
used by 

Spatial Modulation over the WFOV. With super FEI, the NFOV high-resolution is at a 
higher resolution than the optical resolution used by Spatial Modulation for WFOV de- 
tection. Super FEI provides the additional resolution needed to recognize a target that was 
detected as a point by Spatial Modulation. 

Foveation is used by human vision to very efficiently perform the two 
important tasks of WFOV surveillance to detect objects of interest of any size without 
detailed recognition capability (by peripheral vision), and NFOV conventional high- 
resolution imaging of an acquired region of interest or target (by foveal vision). SMI per- 
forms an analogous function to that of peripheral vision - sensing of high-resolution de- 
tail for cueing of unresolved point targets, without detailed point-by-point imaging as 
would be needed for recognition. Foveated SMI adds the foveal component (either as 
pseudo-foveation or true foveation) that allows a detected target to be examined in detail 
in a conventional high-resolution image of the target and/or its immediate surroundings. 

The benefit of foveated SMI (or FEI) is that the total function of WFOV acquisi- 
tion of an unresolved point target and NFOV recognition of the target can be performed 
with may fewer pixels (detector elements) than a conventional high-resolution sensor 
with no SMI. 

An illustrative embodiment of this invention provides a foveal enhancement of 
the baseline spatial-modulation staring sensor. In this enhancement the WFOV low- 
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resolution conventional image is displayed to the operator, but with a selectable subarea 
of that image replaced by a conventionally generated high-resolution image at the same 
scale. The operator would be able to naturally select the region for foveal enhancement 
by simply pointing directly at the detail of special interest which would then be displayed 
to the user at high-resolution. The remainder of the scene would be displayed at low- 
resolution, typically with marks at the locations of point sources detected by the spatial 
modulation. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention description below refers to the accompanying drawings, of which: 

Fig. 1, already described, is schematic diagram of a spatial-modulation staring 
sensor system that can be used to implement the present invention's teachings; 

Fig, 2, already described, is a diagram showing the relationship between reticle 
cells and array elements; 

Fig. 3, already described is a block diagram showing a processing train for proc- 
essing the detector array output of Fig. 2 into detection results; 

Fig. 4A is a graph showing cancellation of clutter according to a prior art sensor 
implementation in the absence of scene motion; 

Fig. 4B is a graph showing a failure to cancel clutter according to a prior art sen- 
sor implementation in the presence of scene motion; 

Fig. 4C is a graph showing successful cancellation of clutter in the presence of 
scene motion using balanced demodulation according to an illustrative embodiment of 
this invention; 

Fig. 5 is a schematic diagram showing an embodiment of the present invention, 
which employs foveal enhancement; 

Fig. 6 is a flow diagram of a foveal enhance procedure according to an illustrative 
embodiment of this invention; and 

Fig. 7 is a schematic diagram showing an illustrative embodiment of the present 
invention that provides Foveal Enhanced imaging by capturing the spillover light in the 
optical transition from the high-resolution image at the reticle plane to the low-resolution 

conventional image at the wide-FOV spatial modulation detector. 
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DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS 



Those skilled in the art and having the benefit of this disclosure will appreciate 
that the descriptions herein are by way of illustration of the principles of the invention in 
each case and not by way of Umitation, 

1 . Balanced Demodulation Procedures 

The function of the demodulation procedure is to derive a spatial-modulation out- 
put image W(xy) in which unresolved point targets are preserved at close to full inten- 
sity, whereas all other scene (i.e., clutter) detail is strongly suppressed. 

With the previously referenced baseline procedure, clutter is strongly suppressed 
when the clutter is not moving relative to the sensor. However, the level of the residual 
clutter increases sharply as a function of relative scene motion. I have recognized that a 
high level of clutter suppression can be achieved despite a certain degree of scene motion 
if one employs a so-termed "balanced demodulation procedure." 

As described above, a previously used type of demodulation function is given by: 

where Vr is the output frame from the FPA on frame number r and Wis the demodulated 
spatial-modulation output frame derived from detector frames 1,2, 8. 

In contrast, one version of a "balanced demodulation function" is given by: 



This is a one-frame-delay version. Another example version is a two-frame-delay 



version: 
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where Vr is the output frame from the FPA on frame number r and Wis the demodulated 
output frame derived from detector frames. 

One of these three demodulation procedures can be selected according to the 
known or expected motion environment in order to maximize the degree of clutter sup- 
pression. The choice may be made in accordance with the following selection table, for 
instance: 

Motion : Demodulation Procedure : 

No motion Baseline 

0 - 0.25 cell width/frame Balanced (One-Frame Delay) 

0.25 - 0.5 cell width/frame Balanced (Two-frame delay) 

The cell width is the angular subtense of a cell in the checkerboard reticle. For 
example, if the reticle has 1,500 x 1,500 cells and covers a 90-degree FOV, the angular 
width of each cell is about 0.06®. If the detector frame rate is 1000 fps, then (for exam- 
ple) 0.5 cell-width/frame corresponds to a target angular rate of 307sec. 

As explained above, the target signal is recovered by taking differences between 
every other frame (e.g., V\ - F3). This differencing cancels the background and recovers 
the reticle-induced sinusoidal temporal modulation of the target signal. In the simple 
one-dimensional case, the target ampUtude is recovered (independent of time /) by taking 
the sum of squares of successive differences (e.g. (Vi- V^f + (F2- ^4)^) + ). 

It can be shown that in the general two-dimensional case the target signal is re- 
covered by taking the sum of squares of four differences: 

This is the baseline spatial-modulation demodulation procedure. But a less com- 
putation-intensive approach, both for the baseline procedure and the balanced-modulation 
procedure, is to take the sum of absolute values rather than the square root of the sum of 
the squares. An example of this approach for the baseline procedure is given by the fol- 
lowing formula, which also implements it as an infinite-impulse-response filter rather 
than a finite-impulse-response filter: 

Wnix^) = ^*^T'„.i(xj;) + (l-^)*|F„(xj;)-F„^^^^ n>l 
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Wi(xo>) = \Vx(x,y)-V^(x,y)\, 

where A: is a parameter such that 0 < ^ < 1 and determines the time constant for the expo- 
nential averaging that the infinite-impulse-response filter implements. 

A similar approach to the balanced-demodulation procedure can be taken, for ex- 
ample, for one-frame and two-frame delays. For a one-frame delay, the following for- 
mula could be used: 

W,{xy) = \V2{x^)-[Vx{xod^V,{xMI2\ 

Here is the corresponding formula for a two-fi'ame delay: 

Wn{xo^) = ^* W,A{xy)^ {\'kY\ Vr,,2{xy) ~ + F„.4(x,>^)]/2|, n > 1 

FFi(xj;) = \V3(xo;)-[Viix^)^Vsix^)V2l 

It may be noted that this preferred means of deriving the spatial-modulation out- 
put image Wn incorporates a number of important features. 

First, it is the modulus, not the square, that is taken of every difference (e.g., 
V\ - V3). In the accumulation of the differences (to form W) the square root is not taken. 
It can be shown that this implementation derives the target signal at almost the same am- 
plitude as by taking the square root of the sum of squares. However this preferred im- 
plementation is computationally more efficient by its elimination of the squaring and 
square root operations. 

Second, since it results fi^om the action of an infinite impulse response (DR) filter, 
the spatial-modulation output Wis an accumulation of the differences (e.g., Vi - F3) over 
time. Being an exponential average, the output approximates an average of the dif- 
ferences (e.g., Fi- V3) over a number of frames. Temporal filtering is necessary in many 
cases because the time response desired in the spatial-modulation output is much longer 
than the FPA frame period. The IIR filter provides the required output time response 
while improving signal-to-noise by temporal averaging. The temporal filtering action is 
performed efficiently and easily on each fi-ame difference as it is developed, and not on 
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the root mean square of four differences as implied in the mathematical baseline deriva- 
tion. 

Finally, the accumulation is taken of all frame differences. In the baseline proce- 
dure only half of the available differences are taken. For example: Fi- F3, V2 - F4, F5 - 
F7, Ve - Kg are taken, but F3 - F5, V4 - Ve, Vy - K9, Fg - Vxo are not. 

To further assist the reader in understanding the principles of balanced demodula- 
tion, reference is made to Figs. 4 A - 4C which show three graphs 140, 142 and 144, re- 
spectively of clutter amplitude C versus frame number. As stated above, the function of a 
Spatial Modulation staring sensor (as shown in Fig. 1) is to detect unresolved hot spot 
point targets in clutter using a much lower resolution detector array (e.g. 256 x 256) than 
would otherwise be required by a conventional staring sensor (e.g. 1, 500 x 1, 500). In 
the Spatial Modulation sensor, a series of consecutive frames VI, V2, V3, is generated by 
the low-resolution detector array (60 in Fig 1). The Spatial Modulation sensor function is 
performed, essentially, by frame-subtraction. 

The basic principle of this frame subtraction process can be understood by con- 
sidering two almost identical output image frames (VI, V3) from the detector array. By 
the action of the moving reticle (30 in Fig 1) a point target that is present in one output 
frame (VI) will not appear in the other frame (V3). These frames (V1,V3) can be sub- 
tracted to form a difference-frame DIF. (DIF = V3-V1). Because the point target is pres- 
ent in only one of these two frames, it is preserved at full amplitude in the difference 
frame (DIF). If the sensor is not moving relative to the scene (graph 140 in Fig. 4 A), all 
other scene detail (i.e. background clutter), being almost the same between the two 
frames (VI, V3), is canceled almost to zero in DIF. The point target can then be readily 
detected in the difference frame DIF because of the very strong attenuation of all back- 
ground clutter. 

However if the sensor is moving relative to the scene (graph 142 in Fig. 4B), the 
clutter will have moved between the taking of the two frames. In that case the frames VI 
and V3 and will then no longer be almost exactly the same. 
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Balanced demodulation is a mechanism for compensating for the effect of scene 
motion so that the clutter can still be subtracted almost to zero. Reference is now made to 
Fig. 4C, which shows the results of balanced demodulation according to the illustrative 
embodiment. With one-frame-delay balanced demodulation the difference (DIF_B) is 
formed as DIF^B = V2 - (VI ^V3)/2. 

If (because of scene motion) the clutter intensity in any particular output detector 
is different in frames VI and V3, casing clutter leakage in the simple difference DIF = 
VI -V3, then this clutter leakage is reduced or attenuated in the balanced demodulation 
difference DIF_B = V2 - (VI +V3)/2. 

For example, assume that, because of scene motion, the output from one particu- 
lar detector element is 12 units in frame VI, 1 1 units in frame V2 and 10 units in frame 
V3. In that case, DIF = V1-V3 = 12-10 = 2 units, whereas DIF^B = V2 - (VI +V3)/2 = 
11- (10+12)/2 = 0. That is, the clutter leakage of 2 units (with prior art demodulation) 
has been reduced to 0 by balanced demodulation. 

The target intensity is also reduced by one-frame balanced demodulation but less 
so. The reticle moves one-half cell per frame. That is, it takes two frames for the reticle 
to block the target. Thus if the nominal target intensity is 100% in frame V2, then it will 
be 50% in frames V1,V3, and therefore 50% in DIFB. Because of the large clutter leak- 
age that can be caused by scene motion the net effect of balanced demodulation with one 
frame delay is maintenance of a strong target-to-clutter ratio with scene motion. 

With balanced demodulation with two-frame delay the basic difference DIF_BB 
is formed as DIFF_BB = V3~(V1^ V4)/2 . 

If the nominal target intensity is 100% in frame V3 then it will be 0% in frames 
VI, V4, and thus nominally 100% in DIF_BB. That is, nominally there is no reduction of 
target signal with balanced demodulation with two-frame delay. 

The point target is made to appear in one frame (e.g. VI) but not the other (e.g. V3) by 
the action of a checkerboard reticle (30 in Fig 1) consisting of alternately clear and 
opaque cells. A high-resolution optical image (10 in Fig 1) of the scene is formed over 
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this reticle which is located within the sensor at an intermediate focal plane (40 in Fig 1) 
and which is moved steadily over the image at a rate of one half checkerboard-cell width 
per frame. After passing through, and being modulated (multiplied) by the reticle, the 
scene image is relayed to a lower resolution detector array (80 in Fig. 2). Suppose that, 
when frame VI is taken, the point target falls within a clear checkerboard cell (76). Then 
when frame V3 is taken the reticle has moved so that it falls over an opaque cell. In that 
case the point target does not appear in frame V3. 

2. Improved Moving Target Detection 

The moving reticle in the Spatial Modulation sensor causes the FPA output signal 
from a point target to be temporally modulated over a sequence of consecutive frames. 
The temporal frequency □ of this modulation is proportional to the reticle spatial fre- 
quency. The amplitude Tof this modulation corresponds to the target signal. The de- 
modulation procedure separates this modulation from the DC background by differencing 
between every other frame — that is, over a phase of 180° — and then recovering the 
modulation amplitude (MOD AMP) by summing the square (or modulus) of these differ- 
ences over multiple frames. (The phase difference per frame is 90°): 



DIFFERENCE^) =DC-^Tsm(a)t-\-7r)-{DC + Tsm6)t) 
= -2T sin cot 



OUTPUT{t) ^DC + Tsin cot (3) 

(4) 

MODAMP = ITyjsm ^ cot + sin ^{(Ot^nll) 

= ITylsin^ (ot + cos^{a)t ^nli) (5) 

= 2r 

For simpUcity of explanation, the target and reticle motions are assumed here to 
be in one (x) direction only. The same principle applies to the general case of x and>' 
motions. 

When the bearing of a point target is stationary relative to the spatial-modulation 
sensor, the demodulated target signal is maximized when the reticle speed is 0.5-cell 
width/frame in x and 0. 125-.cell width/frame 'my. Target motion changes the modulation 
frequency so that the differencing is no longer over 180*^, and the amplitude of the differ- 
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ence is then correspondingly decreased. That is, the output target signal is decreased. 
Target signal is also reduced by motion smearing of the target image. 

In many cases of target motion, the motion of the target is known or at least ap- 
proximately known: for example, a target seen from an aircraft that is executing a roll. 
The roll motion of the aircraft causes relative target motion at the same angular rate, but 
the roll motion of the aircraft can generally be sensed by devices on the aircraft. The 
measured aircraft roll motion can be apphed to the Spatial Modulation sensor to negate 
the effect of that motion. 

For angular rates of target motion up to 0.5 cell width/frame the effect of target 
motion can be negated by changing the reticle speed so that the motion of the target over 
the reticle is restored to its design value of 0,5 cell width per frame in x, and 0. 125 cell 
width/frame iny. For example, suppose it is known that the relative target motion is 
equivalent to 0 .2 cell width in the forward direction of the reticle. In that case, to com- 
pensate for this motion the forward speed of the reticle would be reduced to 0.3 cell 
width/frame and the reverse speed increased to 0.7 cell width per frame. With this pat- 
tern of reticle speeds the relative motion of the target over the reticle would be restored to 
0.5 cell width/frame in both the forward and reverse directions. That is, the target signal 
would be preserved at fiill ampUtude in spite of the target motion. 

For target motion at angular rates above 0.5-cell width per frame, this target- 
motion compensation can be applied only in one (forward) direction of the reticle motion. 
In this case the reticle should be moved as quickly as possible in the other (backward) 
direction and no target information is derived. Target information is derived with full 
target motion compensation when the reticle is moving at its compensated speed in the 
forward direction. Although target information is not available at all times for target mo- 
tions greater than 0.5 cell width/frame, the duty cycle will never be less than 50%. Since 
the raw demodulation procedure output will generally be temporally filtered (e.g., by an 
IIR filter), the effect of this duty cycle will generally be a loss of target signal (of no more 
than 50%) rather than a temporal interruption of signal. 

For scene angular rates above 0.5 cell- width/frame, clutter suppression is de- 
graded. However, with this magnitude of motion the clutter itself will be degraded by 
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motion smearing, thereby partially offsetting the reduced degree of clutter suppression. 
Thus target motion compensation for scene motions above 0.5 cell-width/degree may of- 
ten provide useful target detection capability under conditions in which the target might 
be severely degraded by motion smearing. The compensated reticle speed effectively 
stabilizes the sensor against the motion of the target. 

3. Reticle-Design Tolerances 

Simulation studies have shown that random cell-to-cell variations of as much 
as 1% in the transmission of reticle clear cells are acceptable. Such reticle variations 
typically increase the standard deviation of the residual clutter by no more than about 
20%. Likewise the variation in area between cells is no more than 1% in an illustrative 
embodiment and the opaque cells are considered 100% or "perfectly" opaque in the spec- 
tral wavelength of interest (IR in this example). The reticle can be manufactured in this 
manner using conventional deposition, etching and photolithography processes. 

4, Back-End Optics 

The spatial-modulation optical system extends continuously from the front-end 
objective to the back-end optics (50 in Fig. 1) that forms the final image on the FPA. We 
have determined that imposing a design constraint on this optical system over and above 
those of the baseline system will result in optimized performance. 

As was explained above, a high-angular-resolution image of the object scene is 
formed on the front (input) surface of the reticle. The diameter of the PSF of the front- 
end optics must be of the order of the reticle-cell width. Achieving high front-end reso- 
lution is very important because it directly impacts the amplitude of the modulation of the 
target signal induced by the reticle motion. That modulation provides the means for de- 
tection of the target. The optical system should therefore be designed, manufactured, and 
tested to provide the specified high-resolution of the front-end optics 

Considerations are different at the FPA plane. In relaying the reticle image onto 
the detector FPA, the lateral demagnification of the back-end optics is made equal to the 
ratio of SMF x reticle cell width to detector pixel pitch. From this baseline-system fea- 
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ture — i.e., this imaging of a plurality of reticle cells onto a single detector pixel — one 
might infer that it would be acceptable for the back-end optical system not to completely 
resolve each reticle cell on the FPA plane. 

And this is in fact true; it is not only acceptable but also preferable. Specifically, 
the performance is optimized if, for a PSF centered on an FPA element, not more 
than 50% of the PSF's energy falls within the central 25% of the element's area. Simu- 
lation has shown that the blurring of the high-resolution reticle pattern onto the FPA 
plays an important role in the minimization of artifacts such as the spatial variation in 
sensitivity over the surface of each detector element. Also because of this blurring, each 
detector element is effectively covered equally by clear and opaque cells whether SMF is 
odd or even. (With high-resolution back-end optics and an SMF of 5, a detector would 
be covered sometimes by twelve clear cells (76 in Fig. 2) and sometimes by thirteen. 
This would result in an extraneous modulation in the output from the FPA.) 

Of course, it is, in most cases, desirable to retain the fiill FPA resolution, so at 
least 50% of the PSF's energy should fall on the FPA element. 

5. Foveal Enhancement 

The spatial-modulation staring sensor allows high-performance point-target de- 
tection in clutter to be performed over a WFOV (e.g., 90°) using a small-format FPA. 
For example, detection performance comparable to that of a conventional staring sensor 
with a 1,500 x 1,500 FPA can be achieved using only a 256 x 256 FPA. The baselme 
spatial-modulation staring sensor also provides a conventional low-resolution output 
WFOV image (e.g. 256 x 256) that can be used by an operator to view the entire scene at 
low-resolution. However, even though the resolution provided by the FPA is low, a high- 
resolution image is formed over the chopping reticle. Foveal enhancement provides a 
means of accessing that high-resolution image to provide the operator with a conven- 
tional high-resolution output image of any designated subarea of the scene. The high- 
resolution image of the subarea can be detected either with a second small-format FPA 
(e.g. 256 X 256), or by sharing the original small-format FPA. 
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The conventional low-resolution WFOV output image from the baseline spatial- 
modulation staring sensor can be combined with the Foveal Enhanced high-resolution 
image of a selected area. This combination provides the user with a continuous image of 
the entire WFOV scene, all at the same scale, but with a designated region of interest 
shown at high-resolution. This combined image might, for example, be formed on the 
screen of a user's helmet mounted display (HMD) with a helmet direction sensor. In this 
case the designated (Foveal Enhanced) high-resolution area (e.g. 15° x 15°) would always 
be shown at the center of the display, and the surrounding area of the display (e.g., up to 
45° X 45°) would be covered with a contiguous low-resolution image of the scene at the 
same scale. The operator would designate the position of the desired high-resolution area 
of the scene by aiming his or her head in that direction. The measurement of helmet di- 
rection would be used to designate the desired scene area. The Foveal Enhanced view 
(e.g. 1 5° X 1 5°) would allow the operator using the HMD to view the region of interest in 
the scene at high-resolution while the contiguous low-resolution background view (e.g., 
out to 45° X 45°) would show the overall scene context. In this way the operator would be 
able to dynamically view a wide area scene very much as he or she would by naked eye 
vision. (Naked eye vision provides clarity only over a small central (foveal) area. The 
lower resolution peripheral vision is very important in providing context and orientation). 

Fig. 5 shows one implementation of foveal enhancement that may be incorporated 
into a version of the baseline sensor shown in Fig. 1 . A slewable beam splitter 88 diverts 
some of the high-resolution (e.g. 1 mr PSF) incoming radiation from the front-end optics 
87 onto a second intermediate focal plane 90, A second small-format FPA 100 (e.g. 
256 X 256) is located on that second focal plane 90. It may be essentially identical to the 
primary FPA 104, but the size of the image in its plane is larger than that of the (demag- 
nified) image in the primary FPA's plane, so it covers a relatively small subarea (e.g., 
15°) and thereby affords relatively fine angular resolution (e.g., 1 milliradian (mr)). The 
reticle 93 is a moving/chopping reticle of the type described generally above (reticle 30, 
for example). It is aligned on the primary intermediate focal plane 95. It transmits the 
image through the back-end optics 97, and whence, to the primary FPA 100. 
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The secondary FPA's (100) output, which is an image of the subarea generated 
conventionally, without the reticle's modulation, is used for the foveal area. The low- 
resolution VID output of the primary FPA 104 is used for the peripheral area, and it can 
be marked with the point-source outputs from that FPA's output. Note that the 
above-described processing circuitry is adapted to receive the input of the secondary FPA 
100, and perform the foveal enhancement procedure 1 10 that is now described with refer- 
ence to the flow diagram of Fig. 6. 

In step 1 12, a high-resolution (1 mr IFOV) optical image of the scene is formed 
(over a 90° FOV) onto an intermediate focal plane 95. In step 1 14, this image is modu- 
lated by a moving checkerboard reticle 93 that performs the essential Spatial Modulation 
action described above. After modulation by the reticle 93, the 90° FOV image is relayed 
at low-resolution onto a (primary) low-resolution (256x256) IRFPA 104 in step 1 16. A 
continuous sequence of image frames is produced in accordance with steps 112, 114 and 
116 (subroutine 117). During the production, either as a batch or in a continuous manner, 
the low-resolution image frames from the primary FPA 104 are transmitted to the de- 
modulator of the processing circuitry. The sequence of image frames from this primary 
IRFPA is demodulated (step 1 18) to create a 90° FOV Spatial Modulation output image 
that shows only 1-mr hot spots. 

If and when a hot spot is detected (decision step 120) in the Spatial Modulation 
output image, it is located in step 122. At this time, the corresponding point of the high- 
resolution optical image is identified. In step 124, a small (15° FOV) region of this high- 
resolution optical image is then relayed, at 1 mr resolution, onto a secondary low- 
resolution (256 X 256) IRFPA 100, In step 126, this secondary IRFPA generates a high- 
resolution (1-mr resolution) conventional image of the selected sub area (15 ° FOV) of 
the original input image that can be displayed at the appropriate position on an output im- 
age display. 

During the procedure 1 10, the sequence of raw image frames from the primary 
IRFPA 104 provides (in accordance with subroutine 1 17) a conventional low-resolution 
(6-mr) image of the scene over a 90° FOV (step 128). This 90° FOV low-resolution (6- 

mr) image can be displayed on the output image display (step 130 and dashed branch 
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132). It provides a low-resolution view of the scene detail surrounding the 15° high- 
resolution (1-mr) part of the display. 

Hence, the above-described Foveal Enhanced imaging system procedure allows a 
hot spot target to be detected at high-resolution, and this target and its immediate sur- 
round can then be examined at high-resolution. This can be accomplished using either 
two low-resolution (256 x 256) IRFPAs (as shown and described), or by sharing one low- 
resolution IRFPA. By way of comparison, if a conventional staring sensor were used to 
detect and image targets, over the same 90° FOV, at the same resolution (1-mr), a much- 
larger 1,500 X 1,500 IRFPA would typically be needed to carry out the detection at the 
desired resolution. 

In another embodiment, the primary and secondary FPA's can be, e.g., 256 x 256 
halves of a 256 x 5 12 array. The radiation reflected by the slewable beam splitter is re- 
layed by conventional optics without demagnification onto half the secondary half, and 
the demagnified version of the image that has passed through the reticle is directed to the 
primary half 

In another embodiment, the beam splitter is dichroic and, for example, directs by 
reflection a long-wave infi-ared (LWIR) image onto to the secondary FPA. It would 
transmit a medium wave IR (MWIR) image onto the chopping reticle for WFOV point 
target detection. 

In yet another embodiment, the spectrally selected (e.g., LWIR) Foveal Enhanced 
image is directed by conventional optics to the original FPA (e.g. 256 x 256), which in 
this case would be a two-color FPA. That is, this FPA provides independent outputs of 
the MWIR image (WFOV point target detection) and of the narrow field of view (NFOV) 
LWIR image. 

Note that another complete set of these variations is made available when the sle- 
wable beam splitter is located below, instead of above, the reticle 93 in Fig. 4. 

Fig. 7 illustrates an illustrative embodiment of foveal enhancement system 200 
that is uniquely tied to spatial modulation imaging (SMI) in that SMI fundamentally in- 
volves a loss of photons in the optical transition from the high-resolution image at the 
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plane of the reticle to the relay optic for imaging onto the small-format FP A. With a little 
added effort and in a variety of configurations, it is possible to capture these lost photons 
and put them to work in the Foveal Enhanced high-resolution image without robbing 
light from the spatial modulation mode. 

In accordance with the system 200 of Fig. 7, Fovea! Enhanced imaging is em- 
ployed in concert with spatial modulation. An image is received through front-end optics 
220 to generate a high-resolution image 210 at the chopping reticle 230 in line with the 
intermediate focal plane 240. In accordance with this embodiment, particular attention is 
given to collecting the spillover light in the spatial modulation channel in the transition 
from the high-resolution image 210 at the reticle 230 to the back end or relay optic 250 
that images the images the reticle 30 onto the spatial modulation primary IR focal plane 
array (IRFPA) 260. The full f/number cone 270 available downstream of the reticle im- 
age plane (intermediate focal plane 240) overfills the secondary aperture stop which is 
the image of the first aperture stop formed at the entrance to the relay optic 250. In the 
illustrative embodiment, this overfill is a fundamental and intended consequence of the 
spatial modulation technique. Throughput considerations for the spatial modulation 
channel from front to back lead to this conclusion. Thus the spillover light from the high- 
resolution image at the reticle plane can be harvested for the Foveal Enhanced imaging 
without negative impact on the spatial modulation channel. 

With further reference to Fig. 7 , the optical train for doing this consists of the 
beamsplitter 280, in this case a mirror or dichroic element with a hole/aperture 282 in the 
center to pass through the spatial modulation channel, a condensing optic 290 providing a 
magnification of unity or greater to carry forward the high-resolution image 210, and a 
slewable (represented by double arrow 305) steering mirror 300, which is controllable in 
both pitch and yaw, to access any region of interest (ROI) in the high-resolution scene at 
the small-format secondary detector array 310. The condensing optic 290 provides 
pseudo-foveation for M=l and super-foveation for M>1. Note the subtense of the high- 
resolution image (320) exceeds the size of the affordable small-format FPA 310. In an 
alternate embodiment, the secondary FPA can be replaced by using a shared portion of 
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the primary FPA 260. Foveal Enhanced imaging is enabled by and a perfect comple- 
ment to the spatial modulation imaging technique. 

It is noted that the drive mechanism or actuator (represented schematically, for 
example by arrows 37 in Fig. 1) for the reticle of any embodiment herein can be imple- 
mented using a variety of mechanical and electro-mechanical devices. One actuator for 
the reticle can be a long stroke drive mechanism to translate the reticle across the full ex- 
tent of the scene image at constant velocity (0.5 cell-width per frame and up to 1200 
frames per sec). Alternative the actuator can be a short stroke oscillatory drive mecha- 
nism to translate the reticle at least four cell widths (8 frames) at constant velocity (0.5 
cell-width per frame and up to 1200 frames per sec) plus tum-around-and-velocity- 
stabilization time at each end of the stroke. Alternative the reticle can be actuated based 
upon an active digital device providing independent control of each cell, including mi- 
cromirror arrays (as described in K.E. Petersen, "Micromechanical Light Modulator Ar- 
ray Fabricated on Silicon," Applied Physics. Letter vol. 3 1, 521, 1977), or based upon 
addressable membrane mirrors (as described in L.J. Hombech, "Deformable Mirror Spa- 
tial Light Modulator," Proceedings of SPEE vol. 1 150, 1989 and C. Warde et.al., "Mem- 
brane Mirror Light Modulator Technology," Proceedings of SPDE vol. 3951, 2000), 
and/or upon pneumatic liquid crystals that achieve the equivalent light modulation rate 
(0.5 cell-width per frame and up to 1200 frames per sec) and cell size. 

Finally, Foveal Enhanced imaging can also be implemented by a second FPA lo- 
cated directly on the reticle surface to intercept the high-resolution image and slewed to 
the desired point in the scene by the reticle drive. 

The foregoing has been a detailed description of illustrative embodiments of the 
invention. Various modifications and additions can be made without departing form the 
spirit and scope thereof For example, the arrangement of processing circuitry for han- 
dling images herein can be varied. Any or the procedures described herein can be carried 
out by hardware, software that includes computer-readable medium program instructions, 
or a combination of hardware and software. Additional optical and electro-optical com- 
ponents such as filters, lenses and the like can be provided to the sensor arrangements 
described herein where appropriate. Further, while a reticle composed of clear and 
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opaque alternating cells in a checkerboard pattern is provided, it is expressly contem- 
plated that any combination of transmissive and non-transmissive (or partially transmis- 
sive) cells in any pattern can be provided to form the chopping reticle. Accordingly, this 
description is meant to be taken only by way of example, and not to otherwise Umit the 
5 scope of this invention. 

What is claimed is: 
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